The present invention relates to an improvement of a neural network circuit for performing a processing of recognizing voice patterns, image patterns and the like.
In recent years, a great interest has been taken to the field of neural networks for information processings. These neural networks are implemented by an electronic circuit by imitating structures of neurons of organism.
Conventionally, various neural network circuits have been proposed. FIG. 15 shows a neural network having a layer structure. The neural network shown in FIG. 15 comprises an input layer and an output layer. The output layer includes a plurality of neurons (four neurons in FIG. 15). A set of image or voice data are input to the input layer, and their results of recognition are output from the output layer. Conventionally, there has been a method for hierarchically superposing neural networks having such a layer structure to form a multi-layer neural network.
In the neural network shown in FIG. 15, the neuron forming the output layer comprises a neuron 1 shown in FIG. 14. The neuron 1 shown in FIG. 14 has an output terminal Y and a plurality of input terminals Xi. Each input terminal Xi is multiplied by a weight coefficient Wi corresponding thereto. Results of the multiplication are added. Results of the addition thus obtained are output. A plurality of numerals and characters are preliminarily caused to correspond to the neurons of the output layer such that an output value (addition result value) of the neuron corresponding to a set of data forming any numeral or character is set to a maximum value in the output layer when the data is input to the input layer. This is performed by optimizing a plurality of weight coefficients Wi of each neuron 1. Accordingly, when inputting a set of particular data, an output value of the neuron corresponding thereto is a result of recognition for the particular data. A process for changing and optimizing the weight coefficients Wi is referred to as neural network learning, and a calculating rule of a change quantity of the weight coefficient Wi is referred to as a learning rule.
Most of the neural network processings are realized by a von Neumann type computer.
FIG. 13 shows an example. A network circuit according to the prior art shown in FIG. 13 comprises an input memory 101, a weight memory 102, a multiplier 103, a selecting circuit 104, an adder 105, a register 106, an output memory 107 and a control circuit 108.
The input memory 101 holds many sets of input data to be given to a neural network. The weight memory 102 stores a weight coefficient (hereinafter referred to as a weight value) corresponding to each set of input data. The output memory 107 holds results of calculation as the neural network.
A detailed structure of the conventional circuit shown in FIG. 13 will be described below by using operation. In the conventional circuit, a start signal is sent to the control circuit 108 so that a processing is performed in response to a clock signal. An address A is given from the control circuit 108 to the input memory 101 so that one input data is read from the input memory 101. An address B is given from the control circuit 108 to the weight memory 102 so that a weight value is read from the weight memory 102. The read weight value corresponds to the input data read from the input memory 101.
The multiplier 103 multiplies the input data read from the input memory 101 and the weight value read from the weight memory 102. A result of the multiplication is output to the adder 105.
The selecting circuit 104 selects a value "0" in response to a switching signal A sent from the control circuit 108 only when performing first calculation in each neuron. Accordingly, the adder 105 adds the value "0" and the result of the multiplication performed by the multiplier 103. A result of the addition performed by the adder 105 is held by the register 106. Then, the selecting circuit 104 selects a value of the register 106. The adder 105 adds the value of the register 106 and the result of the multiplication performed by the multiplier 103. A result of the addition is held by the register 106.
At the end of the calculation performed in a first neuron, an address "0" is output from the control circuit 108. At this time, a result of cumulative addition of the data held by the register 106, that is, input data on the first neuron of the output layer of the neural network shown in FIG. 15 is written to the address "0" of the output memory 107. Then, the same addition processing is performed for second and subsequent neurons. Results of the addition are sequentially written to an address "1" and subsequent addresses of the output memory 107.
If image or voice data are given as the input data of the conventional network circuit shown in FIG. 13, images or voices can be recognized by execution of the processing.
Learning of the weight value will be described below. The learning of the weight value includes initial learning for changing the weight value in the initial state and additional learning for additionally changing the weight value which has been initially learned. The initial learning serves to recognize ten numerals 0 to 9 of types "0" and "1" shown in FIG. 16, for example. More specifically, if the number of the neurons of the output layer is 10, the numerals are caused to correspond to the neurons one by one. Each weight value is changed in such a manner that an output of a neuron corresponding to a particular numeral is a maximum value in all the neurons of the output layer when inputting the same numeral. The additional learning performs a processing of changing each weight value in such a manner that each numeral of a type "2" shown in FIG. 16 is additionally recognized individually or locally, for example.
The initial learning and the additional learning perform the same processing of changing the weight value. The processing of changing the weight value in the initial learning will be described below. The case where the numerals 0 to 9 of the types "0" and "1" shown in FIG. 16 are recognized will be described below by using a Hebb learning rule as the learning rule. The Hebb learning rule performs a processing of selecting particular one or more neurons to be learned and changing a plurality of weight values of the neurons. By this processing, a weight value corresponding to each input terminal of the neuron is increased in proportion to a data size Xi sent to the input terminal. A weight value W is obtained by the following equation: EQU W(t+1)=a*Xi+W(t)
wherein "t" represents a time (number of times) and "a" represents a coefficient having a predetermined value of the learning. Each numeral comprises a set of data having a lot of pixels as shown by the numeral "0" in FIG. 18, for example. The pixels have multi-values.
In a learning flow shown in FIG. 17, variables "flag", "type" and "number" to be used for the learning are initialized into "0" at Step S0. Then, a value of each pixel of a first numeral 0 (number=0) of the type "0" shown in FIG. 16 is input to the input layer at Step S1, and an output value of each neuron of the output layer is obtained at Step S2. At Step S3, it is decided whether or not a neuron having a maximum output value corresponds to the numeral "0" (that is, a first neuron). If not so, a weight value of the first neuron is learned at Step S4 where a plurality of weight values W are calculated and changed by the above-mentioned equation and "flag=1" is set to display "learning".
Then, it is decided whether or not the numerals 0 to 9 have completely been recognized at Step S5. If not so, the number of an intended numeral is increased by "1" at Step S6 and the above-mentioned processing is continued.
If all the numerals of the type "0" are completely recognized, it is decided whether the type is "1" or not at Step S7. At first, the type is "0". At Step S8, the type is set to "1" and the number of the input numeral is set to "0" again. Steps S1 to S6 are repeated to recognize all the numerals 0 to 9 with type=1.
If all the numerals of the type "1" are completely recognized, a value of the flag is decided at Step S9. If the flag is "1", learning is performed. Therefore, in order to confirm that the numerals corresponding to the neurons other than the neuron having the weight value changed are always recognized precisely, all the variables "flag", "type" and "number" are set to "0" again at Step S10 and the above-mentioned processing is repeated. After performing such a processing, it is apparent that all the numerals of all the types have precisely been recognized if the flag is 0 at Step S9. Consequently, the processing is ended.
In the conventional neural network circuit, however, a large quantity of image data and the like are input. In order to recognize the data, accordingly, the weight memory should store a great number of weight values and becomes large-sized. For this reason, it has been proposed that the weight memory is formed by a ROM which is physically smaller than a RAM to reduce a size thereof. However, since the weight value cannot be changed by the ROM, it cannot be additionally learned.
If the weight memory is formed by the RAM such that the weight value can be additionally learned, it becomes large-sized.
Consequently, the weight memory should be provided outside. As a result, it takes a lot of time to additionally learn the weight value.